Attention Mechanism:🔗

The attention mechanism is designed to focus on different parts of the input data, depending on the context. In the context of the Transformer model, the attention mechanism allows the model to focus on different words in the input sequence when producing an output sequence. The strength of the attention is determined by a score, which is calculated using a query, key, and value.

The Basic Attention Equation:🔗

Given a Query ($Q$), a Key ($K$), and a Value ($V$), the attention mechanism computes a weighted sum of the values, where the weight assigned to each value is determined by the query and the corresponding key.

The attention score for a query $Q$ and a key $K$ is calculated as:

This score is then passed through a softmax function to get the attention weights:

Where $d_k$ is the dimension of the key vectors (this scaling factor helps in stabilizing the gradients).

Finally, the output is calculated as a weighted sum of the values:

2. Matrix Calculation of Self-Attention:🔗

In practice, we don’t calculate attention for a single word, but rather for a set of words (i.e., a sequence). To do this efficiently, we use matrix operations.

Step 1: Calculate Query, Key, Value matrices🔗

Given an input matrix $X$ (which consists of embeddings of all words in a sequence), and weight matrices $W_Q$, $W_K$, and $W_V$ that we've trained:

Step 2 to 5: Compute the Output of Self-Attention Layer🔗

Given the matrices $Q$, $K$, and $V$ that we've just computed:

2. Calculate the dot product of $Q$ and $K^T$ to get the score matrix:

3. Divide the score matrix by the square root of the depth $d_k$:

4. Apply the softmax function to the scaled score matrix:

5. Multiply the attention weights by the value matrix $V$:

This output is the result of the self-attention mechanism for the input sequence.

---

Multi-Head Attention:🔗

In multi-head attention, the idea is to have multiple sets of Query, Key, Value weight matrices. Each of these sets will generate different attention scores and outputs. By doing this, the model can focus on different subspaces of the data.

Let's denote the number of heads as $h$.

1. For each head $i$, we have its own weight matrices: $W_{Q_i}$, $W_{K_i}$, and $W_{V_i}$.

2. For each head $i$, compute the Query, Key, and Value matrices just like in single-head attention calculate for each head from i to h.

3. Using the $Q_i$, $K_i$, and $V_i$ matrices, we calculate the output for each head:

Now, after obtaining the output for each head, we need to combine these outputs to get a single unified output.

4. Concatenation & Linear Transformation: The outputs from all heads are concatenated and then linearly transformed to produce the final output:

Where $W_O$ is another trained weight matrix.

This multi-head mechanism allows the Transformer to focus on different positions with different subspace representations, making it more expressive and capable of capturing various types of relationships in the data.

---